Electronic device and method of controlling the same

ABSTRACT

An electronic device is provided comprising a camera that obtains an image, and a controller that provides a first control right to a first object included in the obtained image, sets a virtual region including the first object and following a location of the first object, and provides a second control right to a second object when the second object enters into the virtual region.

BACKGROUND

1. Field

Embodiments of this document are directed to an electronic device, andmore specifically to an electronic device that may activate a userinterface using a plurality of objects and a method of controlling theelectronic device.

2. Related Art

Diversified functions of terminals, such as personal computers, laptopcomputers, or mobile phones call for multimedia players with variousfunctions including, for example, still or motion image capturing,playback of music or movie files, game play, or broadcast receipt.

Such terminals may be classified into mobile terminals and stationaryterminals according to mobility and may be classified into handheldterminals and vehicle mount terminals according to portability.

To enforce and add terminal functions, it may be considered to improvestructures and/or software of terminals.

SUMMARY OF THE INVENTION

An exemplary embodiment of this document provides an electronic deviceand a method of controlling the electronic device, which allow a user toeasily and rapidly activate a gesture user interface using a pluralityof objects.

For example, the electronic device and the method of controlling theelectronic device allow user(s) to easily and rapidly activate a gestureuser interface using a plurality of body parts of one or more users.

An exemplary embodiment of this document provides an electronic deviceand a method of controlling the electronic device, which allows a userto easily and rapidly control a control right provided to apredetermined object through a user interface.

The embodiments of this document are not limited thereto, and otherembodiments may also be apparent to those of ordinary skill in the artfrom the description taken in conjunction with the accompanyingdrawings.

According to an embodiment of this document, there is provided anelectronic device including a camera that obtains an image, and acontroller. The controller provides a first control right to a firstobject included in the obtained image, sets a virtual region includingthe first object and following a location of the first object, andprovides a second control right to a second object when the secondobject enters into the virtual region.

According to an embodiment of this document, there is provided anelectronic device including a camera that obtains an image, and acontroller. The controller provides a first control right to a firstobject included in the obtained image, sets a fixed region including thefirst object as a virtual region at a predetermined time, and provides asecond control right to a second object when the second object entersinto the virtual region.

According to an embodiment of this document, there is provided a methodof controlling an electronic device including obtaining an image,providing a first control right to a first object included in theobtained image, setting a virtual region including the first object andfollowing a location of the first object, and providing a second controlright to a second object when the second object enters into the virtualregion.

According to an embodiment of this document, there is provided a methodof controlling an electronic device including obtaining an image,providing a first control right to a first object included in theobtained image, setting a fixed region including the first object as avirtual region at a predetermined time, and providing a second controlright to a second object when the second object enters into the virtualregion.

According to the embodiments of this document, the electronic device andthe method of controlling the electronic device may easily and rapidlyactivate a gesture user interface using a plurality of objects and maycontrol the electronic device through the gesture user interface.

Further, the electronic device and the method of controlling theelectronic device allow a user to easily and rapidly control a controlright provided to a predetermined object through a gesture userinterface.

BRIEF DESCRIPTION OF THE DRAWINGS

The implementations of the embodiments of the present invention will bedescribed in detail as examples with reference to the following drawingsin which like numerals may refer to like elements:

FIG. 1 is a block diagram illustrating an electronic device according toan embodiment of this document;

FIG. 2 is a flowchart illustrating a method of controlling an electronicdevice according to an embodiment of this document;

FIG. 3 illustrates an example of providing the first control right tothe first object according to the method illustrated in connection withFIG. 2;

FIG. 4 illustrates an example of providing the first control right tothe first object according to the method illustrated in connection withFIG. 2;

FIG. 5 illustrates an example of providing the second control right tothe second object and an example where a virtual region including thefirst object follows the first object according to the methodillustrated in connection with FIG. 2;

FIG. 6 illustrates an example of providing the second control right tothe second object according to the method illustrated in connection withFIG. 2;

FIG. 7 illustrates an example of displaying the first and second objectsand the virtual region on the electronic device in the middle ofperforming the method illustrated in connection with FIG. 2;

FIG. 8 illustrates that the electronic device outputs a result ofactivation of a gesture user interface according to the methodillustrated in connection with FIG. 2;

FIG. 9 is a flowchart illustrating a method of controlling an electronicdevice according to an embodiment of this document;

FIG. 10 illustrates an example where a virtual region is formed around afirst body part of a user granted a first control right and a secondbody part of the user enters into the virtual region;

FIG. 11 illustrates an exemplary gesture of the user zooming in thescreen while a both-hand gesture user interface is activated;

FIG. 12 illustrates an exemplary gesture of the user zooming out thescreen while a both-hand gesture user interface is activated;

FIG. 13 is a flowchart illustrating a method of the electronic devicerecognizing a user's zoom-in gesture while a both-hand gesture userinterface is activated;

FIG. 14 is a flowchart illustrating a method of the electronic devicerecognizing a user's zoom-out gesture while a both-hand gesture userinterface is activated;

FIG. 15 illustrates an exemplary user's gesture of zooming in the screenwhile a gesture user interface using the thumb and index finger ofuser's right hand is activated;

FIG. 16 illustrates an exemplary user's gesture of zooming out thescreen while a gesture user interface using the thumb 133B and indexfinger of user's right hand is activated;

FIG. 17 illustrates an example where a body part of a second user entersinto a virtual region set around a first body part of a first user whichis granted a first control right;

FIG. 18 illustrates an example where the first and second users make azoom-in gesture when a control right is additionally granted the lefthand of the second user while the control right is granted the left handof the first user;

FIG. 19 illustrates an example of a zoom-out gesture by the first andsecond users in a case where a control right is additionally provided tothe left hand of the second user while a control right has been providedto the left hand of the first user;

FIG. 20 illustrates a process of determining whether first and secondbody parts are included in the same user in the method shown in FIG. 9;

FIG. 21 is a flowchart illustrating a method of controlling anelectronic device according to an embodiment of this document;

FIG. 22 illustrates an example where a result of inactivation of agesture user interface is displayed on the electronic device 100according to the method illustrated in FIG. 21;

FIG. 23 illustrates a method of controlling an electronic deviceaccording to an embodiment of this document;

FIG. 24 illustrates an example of a control right abandonment gestureaccording to the method illustrated in FIG. 23;

FIG. 25 illustrates an example of displaying on the display of theelectronic device a result of abandonment of a control right accordingto the control right abandonment gesture illustrated in FIG. 24;

FIG. 26 illustrates an example of a control right abandonment gestureaccording to the method illustrated in FIG. 23;

FIG. 27 illustrates an example of displaying on the display of theelectronic device a result of abandonment of a control right accordingto the control right abandonment gesture illustrated in FIG. 26;

FIG. 28 is a flowchart illustrating a method of controlling anelectronic device according to an embodiment of this document;

FIG. 29 illustrates an example of a virtual region setting gestureaccording to the method illustrated in FIG. 28;

FIG. 30 illustrates a process of changing control rights of a gestureuser interface according to the method illustrated in FIG. 28;

FIG. 31 illustrates an example of displaying a result of change ofcontrol rights of the gesture user interface by the gesture illustratedin FIG. 30;

FIG. 32 illustrates an example where if a both-hand gesture userinterface is activated, a region including a virtual region set aroundthe left hand granted a control right is set as a gesture recognitionregion;

FIG. 33 illustrates an example where when a both-hand user interface isactivated, a region including a virtual region set around a left handgranted a control right is set as a gesture recognition region; and

FIG. 34 illustrates an example where when a user's body part granted acontrol right departs from a recognition range of the camera, thecontrol right granted to the body part is lost.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Embodiments of the present invention will now be described more fullywith reference to the accompanying drawings, in which exemplaryembodiments of this document are shown. This document may, however, beembodied in many different forms and should not be construed as beinglimited to the embodiments set forth herein; rather, there embodimentsare provided so that this disclosure will be thorough and complete, andwill fully convey the concept of this document to those skilled in theart.

Hereinafter, an electronic device relating to this document will bedescribed below in more detail with reference to the accompanyingdrawings. In the following description, suffixes “module” and “unit” aregiven to components of the mobile terminal in consideration of onlyfacilitation of description and do not have meanings or functionsdiscriminated from each other.

The electronic device described in the specification may be a mobileterminal such as a cellular phone, a smart phone, a laptop computer, adigital broadcasting terminal, personal digital assistants (PDA), aportable multimedia player (PMP), a navigation system and so on. Also,the electronic device may be a stationary device such as a television, apersonal computer, and so on. FIG. 1 is a block diagram of an electronicdevice according to an embodiment of this document.

As shown, an electronic device 100, such as, for example, a displaydevice, may include a communication unit 110, a user input unit 120, anoutput unit 150, a memory 160, an interface 170, a controller 180, and apower supply 190. Not all of the components shown as shown in FIG. 1 maybe essential parts and the number of components included in theelectronic device 100 may be varied.

The communication unit 110 may include at least one module that enablescommunication between the electronic device 100 and a communicationsystem or between the electronic device 100 and another device. Forexample, the communication unit 110 may include a broadcasting receivingmodule (broadcasting receiver) 111, an Internet module 113, and a nearfield communication module 114.

The broadcasting receiving module 111 may receive broadcasting signalsand/or broadcasting related information from an external broadcastingmanagement server through a broadcasting channel.

The broadcasting channel may include a satellite channel and aterrestrial channel, and the broadcasting management server may be aserver that generates and transmits broadcasting signals and/orbroadcasting related information or a server that receives previouslycreated broadcasting signals and/or broadcasting related information andtransmits the broadcasting signals and/or broadcasting relatedinformation to a terminal. The broadcasting signals may include not onlyTV broadcasting signals, radio broadcasting signals, and databroadcasting signals but also signals in the form of a combination of aTV broadcasting signal and a radio broadcasting signal of a databroadcasting signal.

The broadcasting related information may be information on abroadcasting channel, a broadcasting program or a broadcasting serviceprovider, and may be provided even through a communication network.

The broadcasting related information may exist in various forms. Forexample, the broadcasting related information may exist in the form ofan electronic program guide (EPG) of a digital multimedia broadcasting(DMB) system or in the form of an electronic service guide (ESG) of adigital video broadcast-handheld (DVB-H) system.

The broadcasting receiving module 111 may receive broadcasting signalsusing various broadcasting systems. The broadcasting signals and/orbroadcasting related information received through the broadcastingreceiving module 111 may be stored in the memory 160.

The Internet module 113 may correspond to a module for Internet accessand may be included in the electronic device 100 or may be externallyattached to the electronic device 100.

The near field communication module 114 may correspond to a module fornear field communication. Further, Bluetooth®, radio frequencyidentification (RFID), infrared data association (IrDA), ultra wideband(UWB) and/or ZigBee® may be used as a near field communicationtechnique.

The user input unit 120 is used to input an audio signal or a videosignal and may include a camera 121 and a microphone 122.

The camera 121 may process image frames of still images or moving imagesobtained by an image sensor in a video telephony mode or a photographingmode. The processed image frames may be displayed on a display 151. Thecamera 121 may be a 2D or 3D camera. In addition, the camera 121 may beconfigured in the form of a single 2D or 3D camera or in the form of acombination of the 2D and 3D cameras.

The image frames processed by the camera 121 may be stored in the memory160 or may be transmitted to an external device through thecommunication unit 110. The electronic device 100 may include at leasttwo cameras 121.

The microphone 122 may receive an external audio signal in a call mode,a recording mode or a speech recognition mode and process the receivedaudio signal into electric audio data. The microphone 122 may employvarious noise removal algorithms for removing or reducing noisegenerated when the external audio signal is received.

The output unit 150 may include the display 151 and an audio outputmodule 152.

The display 151 may display information processed by the electronicdevice 100. The display 151 may display a user interface (UI) or agraphic user interface (GUI) relating to the electronic device 100. Inaddition, the display 151 may include at least one of a liquid crystaldisplay, a thin film transistor liquid crystal display, an organiclight-emitting diode display, a flexible display and a three-dimensionaldisplay. Some of these displays may be of a transparent type or a lighttransmissive type. That is, the display 151 may include a transparentdisplay. The transparent display may include a transparent liquidcrystal display. The rear structure of the display 151 may also be of alight transmissive type. Accordingly, a user may see an object locatedbehind the body of terminal through the transparent area of the terminalbody, occupied by the display 151.

The electronic device 100 may include at least two displays 151. Forexample, the electronic device 100 may include a plurality of displays151 that are arranged on a single face at a predetermined distance orintegrated displays. The plurality of displays 151 may also be arrangedon different sides.

Further, when the display 151 and a sensor sensing touch (hereafterreferred to as a touch sensor) form a layered structure that is referredto as a touch screen, the display 151 may be used as an input device inaddition to an output device. The touch sensor may be in the form of atouch film, a touch sheet, and a touch pad, for example.

The touch sensor may convert a variation in pressure applied to aspecific portion of the display 151 or a variation in capacitancegenerated at a specific portion of the display 151 into an electricinput signal. The touch sensor may sense pressure of touch as well asposition and area of the touch.

When the user applies a touch input to the touch sensor, a signalcorresponding to the touch input may be transmitted to a touchcontroller. The touch controller may then process the signal andtransmit data corresponding to the processed signal to the controller180. Accordingly, the controller 180 can detect a touched portion of thedisplay 151.

The audio output module/unit 152 may output audio data received from theradio communication unit 110 or stored in the memory 160. The audiooutput module 152 may output audio signals related to functions, such asa call signal incoming tone and a message incoming tone, performed inthe electronic device 100.

The memory 160 may store a program for operation of the controller 180and temporarily store input/output data such as a phone book, messages,still images, and/or moving images. The memory 160 may also store dataabout vibrations and sounds in various patterns that are output fromwhen a touch input is applied to the touch screen.

The memory 160 may include at least one of a flash memory, a hard disktype memory, a multimedia card micro type memory, a card type memory,such as SD or XD memory, a random access memory (RAM), a static RAM(SRAM), a read-only memory (ROM), an electrically erasable programmableROM (EEPROM), a programmable ROM (PROM) magnetic memory, a magnetic diskor an optical disk. The electronic device 100 may also operate inrelation to a web storage performing the storing function of the memory160 on the Internet.

The interface 170 may serve as a path to all external devices connectedto the electronic device 100. The interface 170 may receive data fromthe external devices or power and transmit the data or power to internalcomponents of the electronic device 100 or transmit data of theelectronic device 100 to the external devices. For example, theinterface 170 may include a wired/wireless headset port, an externalcharger port, a wired/wireless data port, a memory card port, a port forconnecting a device having a user identification module, an audio I/Oport, a video I/O port, and/or an earphone port.

The controller 180 may control overall operations of the electronicdevice 100. For example, the controller 180 may perform control andprocessing for voice communication. The controller 180 may also includean image processor 182 for pressing image, which will be explainedlater.

The power supply 190 receives external power and internal power andprovides power required for each of the components of the electronicdevice 100 to operate under the control of the controller 180.

Various embodiments described in this document can be implemented insoftware, hardware or a computer readable recording medium. According tohardware implementation, embodiments of this document may be implementedusing at least one of application specific integrated circuits (ASICs),digital signal processors (DSPs), digital signal processing devices(DSPDs), programmable logic devices (PLDs), field programmable gatearrays (FPGAs), processors, controllers, micro-controllers,microprocessors, and/or electrical units for executing functions. Theembodiments may be implemented by the controller 180 in some cases.

According to software implementation, embodiments such as procedures orfunctions may be implemented with a separate software module executingat least one function or operation. Software codes may be implementedaccording to a software application written in an appropriate softwarelanguage. The software codes may be stored in the memory 160 andexecuted by the controller 180.

Hereinafter, a method of controlling an electronic device according toan embodiment of this document will be described in connection withFIGS. 1 and 2.

FIG. 2 is a flowchart illustrating a method of controlling an electronicdevice according to an embodiment of this document.

The controller 180 of the electronic device 100 analyzes an imagecaptured by the camera 121 (S110). Specifically, the image processor 182of the controller 180 analyzes the captured image to trace the locationof an object such as each body part of a user.

For example, the image processor 182 senses a skin tone of the userpositioned in front of the camera 121 area to recognize the user's bodypart corresponding to the skin tone and to trace the location of thebody part. In general, skin tones of human face and hands are exposed asis. Further, a human face has a left and right symmetrical shape, andboth eyes are relatively dark in general. The user's face may berecognized by such characteristic color patterns. Further, an objectthat is relatively freely movable within a predetermined radius of theface and has a smaller size than that of the face may be considered as ahand. A hand may be considered to have a lower skin tone when making afist than when being opened. Further, a case where a portion consideredas a hand alternately includes skin tone zones and non-skin tone zonesmay be considered as when the hand is opened.

The above image analysis method of recognizing a user's body part bysensing his or her skin tone and of tracing the location of the bodypart is merely an example of various image analysis methods that may beperformed by the image processor 182. The embodiments of this documentare not limited thereto, and other various image analysis methods may becarried out by the image processor 182.

For example, the controller 180 may trace trajectories of objectsincluded in the obtained image and may analyze whether the trajectoriesform a predetermined trajectory. In a case where a user is included inthe obtained image, the controller 180 may trace a location of a user'sbody part to recognize a specific gesture of the user.

The camera may be integrally provided with the electronic device 100 tocapture an external object or may be provided separately from theelectronic device 100 to transfer captured images of objects to theelectronic device 100. According to an embodiment, the camera 121 mayinclude a 2D camera, a 3D camera, or a combination thereof. Further,even though a single camera is shown in each of the figures forconvenience of description, a plurality of cameras may also be provided.

When the controller 180 analyzes an image, an analysis of a 2D imagecaptured by a 2D camera may be performed faster than an analysis ofdepth data captured by a 3D camera (hereinafter, referred to as “3Dimage”). However, analysis results of the 2D image show someshortcomings, such as being affected a lot by surrounding lights, adifficulty recognizing colors when surrounding lights are changed, or ashadow of an object being determined to be the same as the object.

The 3D image analysis may overcome such shortcomings of the 2D imageanalysis, but may cause an increase in data throughput and as a resultreduced data processing rate.

3D cameras capable of 3D capturing may include, but not limited to,stereoscopic cameras, structural light cameras, and TOF (Time of Flight)cameras.

A stereoscopic camera may obtain a stereoscopic image by using twocapturing lenses with a predetermined distance therebetween. The twolenses interoperate with each other in focus and exposure. Thestereoscopic camera allows an object to be viewed as a stereoscopicimage due to binocular parallax between the two lenses. A structurallight camera radiates a infrared ray to an object and analyzes ascattering pattern of the reflected infrared ray to obtain informationon a depth to the object.

A TOF camera obtains a stereoscopic image by measuring a time that itwould take a light beam radiated to an object to come back. For example,as an object is away from the camera, a travelling time of a light beamincreases, and as the object is close to the camera, the travelling timeof the light beam decreases. Thus, calculation of the travelling timesof light beams may measure distances to the object, and combination ofthe measured distances may obtain a stereoscopic image. The TOF cameramay be provided with an illumination device for illuminating the objectwith light.

In the electronic device 100 according to the embodiments of thisdocument, a 2D image obtained by a 2D camera may be analyzed to increaseimage analysis speed, and a 3D image obtained by a 3D camera may beanalyzed to increase accuracy of image analysis. According to anembodiment, a 2D image analysis and a 3D image analysis may be bothperformed on a 2D image and a 3D image.

When an analysis result of the obtained image shows that a trajectory ofa first object is recognized as a predetermined control right obtainingtrajectory, the controller 180 provides the first object with a controlright (S120). Then, the controller 180 analyzes an image obtained by thecamera 121 to trace the trajectory of the first object. When thetrajectory of the first object corresponds to a predeterminedtrajectory, a function corresponding to the predetermined trajectory maybe performed.

According to embodiments, the first object may include, but not limitedto, a users' body part, such as an arm, a leg, a hand, a foot, or afinger, or a device owned by the user, such as a remote controller.

If the first object is a certain body part of the user, the controller180 providing the body part with the first control right preferablymeans the controller 180 is activating a gesture user interface usingthe specific body part. In particular, if the body part is one of bothhands, the provision of the first control right activates a one-handuser interface for the electronic device 100.

For example, if the electronic device 100 is a display device, the usermay perform various functions, such as volume adjustment, channeladjustment, zoom-in, zoom-out, or light-and-shade adjustment for thedisplay device by making a predetermined gesture with a hand or otherbody part or object provided with the control right.

When the gesture user interface using the user's specific body part isactivated, the user may manipulate the electronic device 100 with agesture made by the specific body part having the control right.Accordingly, the user may use the electronic device 100 moreconveniently than when manipulating the electronic device 100 by usingbuttons on the electronic device 100 or a remote controller.

FIG. 3 illustrates an example of providing the first control right tothe first object according to the method illustrated in connection withFIG. 2. It is assumed as shown in FIG. 3 that the first object is theleft hand 132 of a user 130. Referring to FIG. 3, the user 130 mayobtain a control right by making a gesture of waving a lower part 131 ofhis left arm in left and right directions with the left hand 132 opened.The gesture may be preset as an operation of providing the control rightto the user 130 between the electronic device 100 and the user 130.Other various gesturers than that shown as shown in FIG. 3 may be set asthe gesture of providing the control right to the user 130.

If the camera 121 is a 2D camera, the controller 180 may analyze a 2Dimage of the user obtained by the camera 121 to recognize the user'sgesture, and if the camera 121 is a 3D camera, the controller 180 mayanalyze a 3D image of the user obtained by the camera 121 to recognizethe user gesture.

FIG. 4 illustrates an example of providing the first control right tothe first object according to the method illustrated in connection withFIG. 2. Referring to FIGS. 3 and 4, the electronic device 100 may be astationary electronic device, such as a TV 100 with a camera 121 or amobile device, such as a mobile terminal 100 with a camera 121.

As shown As shown in FIG. 4, when the user 130 changes gestures from (a)making a fist using his right hand 133 to (b) opening an index finger133A with the other fingers of the right hand closed, the electronicdevice 100 may provide the index finger 133A of the user's right hand133 with a control right to control the operation of the electronicdevice 100. For example, it is assumed as shown in FIG. 4 that the firstobject is the index finger 133A of the user's right hand 133. Asdescribed earlier, other various gestures than that shown as shown inFIG. 4 may be set as the gesture of providing the user 130 with thecontrol right.

When the first object is granted the first control right, the controller180 sets a virtual region that includes the first object and follows alocation of the first object (S130). The phrase “follows a location ofthe first object” preferably means that when the first object is moved,the virtual region including the first object is also moved followingthe first object.

The virtual region may be a 3D space including the first object, and mayhave various shapes, such as a sphere or a rectangular parallelepiped.For example, the camera 121 used for obtaining an image may be a 3Dcamera that may obtain depth data for an external object. The virtualregion may be a 2D region including the first object as viewed from thecamera 121 and may have various shapes, such as a circle or a rectangle.The camera 121 used for obtaining an image may be a 2D that may obtain a2D image for an external object.

When the virtual region is set, the controller 180 determines whether asecond object enters into the virtual region (S140). If the secondobject enters into the virtual region, the controller 180 provides asecond control right to the second object (S150). If the second objectis a predetermined body part of the user, the controller 180 providingthe second control right to the predetermined body part preferably meansthe controller 180 is activating a gesture user interface by thatpredetermined body part.

The controller 180 may provide the control right to the second objectwhile maintaining the first control right provided to the first object.If the first object is one of the user's two hands and the second objectis the other hand of the same user, this preferably means that thecontroller 180 activates both-hand user interface for the electronicdevice 100. That is, the user can use the gestures made by his/her bothhands having the control rights to control the operation of theelectronic device 100.

As a variation, if the first object is a predetermined body part of afirst user and the second object is a predetermined body part of asecond user different from the first user, this preferably means thatthe controller 180 adds the second user as a person authorized tocontrol the electronic device 100 in addition to the first user. Then,the first and second users both may exercise the control rights on theelectronic device 100 and use the gestures to control the electronicdevice 100 at a given time.

For example, if the electronic device 100 is a display device, the usermay perform various functions, such as zoom-in, zoom-out, volumeadjustment, channel adjustment, or light-and-shadow adjustment for thedisplay device by making a predetermined gesture using both hands of thesame user or different users granted the control right.

If the electronic device 100 is a surveillance camera that captures andrecords a certain area, the user may magnify or reduce an area to becaptured by making a predetermined gesture using his both hands and maycontrol a process of playing a recorded image.

In one example, where the second control right is provided to the secondobject, the controller 180 may inactivate or relinquish the firstcontrol right provided to the first object while the second controlright is provided to the second object. For example, if the first objectis one of user's both hands and the second object is the other hand ofthe user, the controller 180 can switch the hand used for the one-handuser interface. In another example, if the first object is a specificbody part of the first user and the second object is a specific bodypart of the second user, the controller 180 can change a personauthorized to control the electronic device 100 from the first user tothe second user.

FIG. 5 illustrates an example of providing the second control right tothe second object and an example where a virtual region including thefirst object follows the first object according to the methodillustrated in connection with FIG. 2. As shown in FIG. 5, the firstobject is the left hand 132 of the user and the second object is theright hand 133 of the same user.

Referring to (a) of FIG. 5, a virtual region which is a spherical 3Dspace and includes the left hand 132 is set around the user's left handhaving obtained the first control right for the electronic device 100,and it can be recognized that the user moves his right hand 133 into thevirtual region.

Referring to (b) of FIG. 5, when the user's right hand 133 is moved andentered into the virtual region, the controller 180 may provide thesecond control right to the right hand 133. The controller 180 mayactivate a both-hand gesture user interface for the electronic device100 by maintaining the control right provided to the left hand 132 ormay inactivate the control right provided to the left hand 132 to changethe hand used for a gesture user interface from the left hand to theright hand.

The controller 180 may set the virtual region from a time point ofprovision of the control right to the left hand 132. Then, aftermaintaining the setting of the virtual region during only apredetermined time period, the controller 180 may release the setting ofthe virtual region.

For example, in a case where the user does not move the right hand 133to the virtual region within the predetermined time period, this maymean that the user does not intend to activate the both-hand gestureuser interface where both hands of the user can control the electronicdevice 100. Also, if the user moves the right hand 133 to the virtualregion to activate the both-hand user interface within the predeterminedtime period, the virtual region may be utilized as a user's interfacingspace. The time of maintaining the setting of the virtual region may bepredetermined upon manufacture of the electronic device 100 ordetermined by the user.

Also, in a case where a trajectory of the left hand 132 is equal to apredetermined trajectory after a control right has been provided to theleft hand 132, that is, when the user makes a predetermined gestureusing the left hand 132, the controller 180 may set a virtual region.Once a predetermined time elapses after the virtual region has been set,the controller 180 may release the setting of the virtual region. Areason for this has been described above.

Referring to (c) of FIG. 5, it can be seen that the virtual regionincluding the left hand 132 follows the movement of the left hand 132while maintaining the same shape. For instance, the virtual region setaround the left hand 132 is moved as the left hand 132 moves. As avariation, unlike that shown in (c) of FIG. 5, the virtual region may beset as a fixed region (in space) including the left hand 132 at the timeof setting the virtual region so that the virtual region may not moveeven though the left hand 132 moves.

FIG. 6 illustrates an example of providing the second control right tothe second object according to the method illustrated in connection withFIG. 2. As shown in FIG. 6, the first object is the index finger 133A ofa user's right hand 133, and the second object is the thumb 133B of thesame user's right hand 133.

Referring to FIG. 6, a virtual region which is, e.g., a spherical 3Dspace and includes the index finger 133A of the right hand 133 is setaround the index finger 133A of the right hand 133 granted the firstcontrol right for the electronic device 100. It can be seen that thethumb 133B of the right hand 133 enters into the virtual region when theuser opens the thumb 133B of his right hand 133.

Then, the controller 180 may provide the second control right to thethumb 133B of the right hand 133. The controller 180 may activate thegesture user interface using the index finger 133A and the thumb 133Bfor the electronic device 100 by maintaining the control right providedto the index finger 133A of the right hand 133 or may activate thegesture user interface using only the index finger 133A by inactivatingthe control right provided to the index finger 133A.

As described above in connection with FIG. 5, the controller 180 mayrelease the setting of the virtual region when a control right isprovided to the thumb 133B after having maintained the virtual regionduring a predetermined time period. Also, as shown in (c) of FIG. 5, thevirtual region including the index finger 133A may follow the movementof the index finger 133A while generally maintaining the same shape ormay also be set as a region whose location is fixed and which includesthe index finger 133A at the time of setting the virtual region.

It has been described in connection with FIG. 6 that the thumb 133B andthe index finger 133A of the user's right hand 133 may be used for thegesture user interface of the electronic device 100 according to anembodiment of this document. However, the embodiments of this documentare not limited thereto. For example, any two fingers of the right hand133, thumb and index finger of the left hand 132, any two fingers of theleft hand 132, or one finger of the left hand 132 and one finger of theright hand 133 may also be used for the gesture user interface of theelectronic device 100 according to an embodiment of this document.

Although not shown in FIG. 2, the controller 180 may display the firstobject granted the first control right, the virtual region set based onthe location of the first object, and the second object approaching thevertical region on the display 151 of the electronic device 100. Thesecond object may be an object preset as a candidate that may obtain acontrol right for the electronic device 100.

For example, the first object and the second object to which thecontroller 180 provides the control right may be a user's body parts. Ifthe controller 180 determines whether to activate the both-hand gestureuser interface for the electronic device 100, the first and secondobjects may be user's both hands. However, the embodiments of thisdocument are not limited thereto. For example, the first and secondobjects may also be other body parts of the user, which have been presetto be capable of being used for obtaining the control right for theelectronic device 100.

FIG. 7 illustrates an example of displaying the first and second objectsand the virtual region on the electronic device in the middle ofperforming the method illustrated in connection with FIG. 2. FIG. 7shows an image displayed on the display 151 of the electronic device 100when the electronic device 100 is a display device, the user's left handis the first object, and the user's right hand is the second object.

Referring to FIG. 7, it can be seen that the controller 180 displays theuser's left and right hands which are the first object and the secondobject, respectively, and a virtual region to which the left and righthands have been entered. Although not shown in FIG. 7, the controller180 may sequentially display steps in which the user's right hand entersinto the virtual region on the display 151.

Also, the controller 180 may display the user's whole body so that theuser's two hands and the virtual region are differentiated from theother body parts. However, the embodiments of this document are notlimited thereto. For example, the controller 180 may display thetrajectory of the user's left hand, whether to obtain the first controlright for the left hand, the virtual region set based on the firstcontrol right, and the trajectory of the user's right hand to obtain thesecond control right on the display 151 in various manners.

According to an embodiment, the controller 180 may display a textmessage notifying whether control rights have been provided to the firstand second objects and whether the control rights have been inactivatedon the display 151 of the electronic device 100 or may output a voicemessage notifying whether the control rights have been provided to thefirst and second objects and whether the control rights have beeninactivated through the audio output module 152 of the electronic device100.

FIG. 8 illustrates that the electronic device outputs a result ofactivation of a gesture user interface according to the methodillustrated in connection with FIG. 2. As shown in FIG. 8, theelectronic device 100 is a display device, and a message notifying thata gesture user interface using the user's two hands has been activatedis displayed.

Referring to FIG. 8, the controller 180 may output a message saying“both-hand gesture UI mode has been activated” through the display 151in the form of a text message or through the audio output module 152 inthe form of a voice message. Although not shown in FIG. 8, when thegesture user interface is inactivated or object used for the gestureuser interface is changed, the controller 180 may output a text or voicemessage corresponding thereto.

It has been described in connection with FIGS. 7 and 8 that messages forthe gesture user interface are displayed on the display 151 assumingthat the electronic device 100 may be a TV. However, such messages mayalso be displayed for any type of electronic device 100, e.g., when theelectronic device 100 is a mobile terminal according to an embodiment ofthis document.

Hereinafter, examples where the controller 180 of the electronic device100 controls setting of a gesture user interface using a specific bodypart of a user of the electronic device 100 will be described.

FIG. 9 is a flowchart illustrating a method of controlling an electronicdevice according to an embodiment of this document. The method ofcontrolling an electronic device will now be described with reference toFIGS. 1 and 9.

The controller 180 of the electronic device 100 analyzes an imageobtained by the camera 121, provides a first control right to a firstbody part of a user (S210), and sets a virtual region following thelocation of the first body part (S220). Then, if an image analysisresult shows that a second body part enters into the virtual region(S230), the controller 180 determines whether the first and second bodyparts belong to the same user (S240).

If the first and second body parts belong to the same user, thecontroller 180 may provide a second control right to the second bodypart. In this case, the controller 180 may maintain the first controlright provided to the first body part (S250). This preferably means thata multi-gesture user interface using any one of the first and secondbody parts is activated for the electronic device 100.

However, unlike step S250 in FIG. 9, as a variation, the controller 180may inactivate the first control right provided to the first body partwhile providing the control right to the second body part. Thispreferably means that a gesture user interface using the second bodypart is activated for the electronic device 100.

If the first and second body parts do not belong to the same user atstep S240, the controller 180 maintains the first control right providedto the first body part but does not provide a control right to thesecond body part (S260). As a variation, however, the controller 180 mayalso provide a control right to the second body part. This preferablymeans that each of a plurality of different users becomes a personauthorized to control the electronic device 100.

Also, still as another variation to step S260 in FIG. 9, the controller180 may inactivate the first control right for the first body part andmay provide the second control right to the second body part. This meansthat a person authorized to control the electronic device 100 is changedfrom the user of the first body part to the user of the second bodypart.

FIG. 10 illustrates an example where a virtual region is formed around afirst body part 132 of a user 130 granted a first control right and asecond body part 133 of the user 130 enters into the virtual region.

Referring to FIG. 10, the first control right is provided to the lefthand 132 of the user 130, a spherical 3D virtual region is formed aroundthe left hand 132, and the right hand 133 of the user 130 has enteredinto the virtual region. Since the left hand 132 and the right hand 133positioned in the virtual region belong to the same user 130, thecontroller 180 provides the second control right to the right hand ofthe user 130. However, it is optional whether to maintain the controlright to the left hand 132.

Depending on whether to maintain the control right to the left hand 132,a gesture user interface mode for the electronic device 100 may become aboth-hand gesture user interface mode or a one-hand gesture userinterface mode. This has been described above, and thus, repetitivedescription thereof will be omitted.

FIG. 11 illustrates an exemplary gesture of the user zooming in thescreen while a both-hand gesture user interface is activated. Referringto FIG. 11, in a case where a distance between his two hands 132 and 133increases by D1-->D2-->D3 by the user opening his two arms from (a)through (b) to (c), the controller 180 of the electronic device 100 mayrecognize this gesture as zooming in the screen. However, this gestureis merely an example of performing a predetermined zoom-in operation,and the embodiments of this document are not limited thereto. Forexample, other various gestures than increasing the distance between thetwo hands may also be used for zooming in the screen.

FIG. 12 illustrates an exemplary gesture of the user zooming out thescreen while a both-hand gesture user interface is activated. Referringto FIG. 12, in a case where a distance between his two hands 132 and 133decreases by D3-->D2-->D1 by the user putting his two arms together from(a) through (b) to (c), the controller 180 of the electronic device 100may recognize this gesture as zooming out the screen. However, thisgesture is merely an example of performing a predetermined zoom-outoperation, and the embodiments of this document are not limited thereto.For example, other various gestures than decreasing the distance betweenthe two hands may also be used for zooming out the screen.

FIG. 13 is a flowchart illustrating a method of the electronic device100 recognizing a user's zoom-in gesture while a both-hand gesture userinterface is activated. It is assumed in FIG. 13 that in the electronicdevice 100, a gesture of increasing a distance between the two hands isset as a zoom-in operation, and a gesture of decreasing the distancebetween the two hands is set as a zoom-out operation. Hereinafter, theelectronic device 100 is assumed to be a TV and the method ofrecognizing the zoom-in gesture will be described with reference to thefigures. However, these are mere examples and the invention fullyencompasses other variations.

If the distance between the user's two hands 132 and 133 increases bythe user opening his arms (S270), the controller 180 of the TV 100analyzes the user's image captured by the camera 121 to recognize avariation in depth to the two hands 132 and 133 and, based on thevariation in depth, determines whether a travelling direction of the twohands 132 and 133 with respect to the TV 100 is toward the screen of theTV 100 (S271).

Here, the “travelling direction of the two hands 132 and 133” may referto a direction in which a middle point between the two hands 132 and 133is moved respective of the screen of the TV 100. However, theembodiments of this document are not limited thereto. For example, thetravelling direction of the two hands 132 and 133 may be an averagetravelling distance of the two hands 132 and 133 with respect to thescreen of the TV 100. This may apply to FIG. 14 as well.

If the travelling direction of the two hands 132 and 133 is toward thescreen of the TV 100, for example when the user makes a gesture ofincreasing the distance between the two hands by stretching his bothhands 132 and 133 frontward, the controller 180 recognizes this as agesture of zooming in the screen and accordingly performs a zoom-inoperation (S272).

However, if the travelling direction of the two hands 132 and 133 is adirection away from the screen of the TV 100, for example when the usermakes a gesture of increasing the distance between his both hands bypulling his hands 132 and 133 back, the controller 180 maintains thescreen as is without recognizing this as a gesture of zooming in thescreen (S273).

When recognizing a user's gesture of reducing the distance between thehands 132 and 133 after the user's gesture of increasing the distancethe hands 132 and 133 has been recognized (S274), the controller 180analyzes the user's image captured by the camera 121 to recognize avariation in depth to the two hands 132 and 133 and, based on thevariation in depth, determines whether a travelling direction of the twohands 132 and 133 with respect to the TV 100 is toward the screen of theTV 100 (S275).

If the travelling direction of the two hands 132 and 133 is toward thescreen of the TV 100, for example when the user makes a gesture ofdecreasing the distance between the two hands by stretching his bothhands 132 and 133 frontward, the controller 180 recognizes this as agesture of zooming out the screen and accordingly performs a zoom-outoperation (S276).

However, if the travelling direction of the two hands 132 and 133 is adirection away from the screen of the TV 100, for example when the usermakes a gesture of decreasing the distance between his both hands bypulling his hands 132 and 133 back, the controller 180 maintains thescreen as is without recognizing this as a gesture of zooming out thescreen (S277).

According to the method of recognizing the zoom-in gesture as shown inFIG. 13, although the user increases the distance between the hands 132and 133 by stretching this both hands 132 and 133 toward the screen andthen pulls back his hands 132 and 133 to decrease the distance betweenthe hands 132 and 133, the controller 180 performs only the zoom-inoperation but not zoom-out operation. This comes from the fact that auser of the electronic device 100 generally makes a zoom-in or zoom-outgesture with constant directivity. According to an embodiment, adirection for determining whether to perform a zoom-in or zoom-outoperation may be set to be opposite to that considered in FIG. 13.

FIG. 14 is a flowchart illustrating a method of the electronic device100 recognizing a user's zoom-out gesture while a both-hand gesture userinterface is activated. In contrast to FIG. 13, it is assumed in FIG. 14that in the case of recognizing a gesture of varying the distancebetween the hands 132 and 133 while the travelling direction of theuser's both hands 132 and 133 is a direction away from the screen of theTV 100, the electronic device 100 performs a zoom-in or zoom-outoperation. Hereinafter, the electronic device 100 is assumed to be a TVand the method of recognizing the zoom-in gesture will be described withreference to the figures. However, these are merely examples and theinvention fully encompasses other variations.

Upon recognizing a gesture of decreasing the distance between the twohands 132 and 133, the controller 180 of the TV 100 determines whetherthe two hands 132 and 133 move toward or opposite to the screen of theTV 100 (S281).

If the travelling direction of the two hands 132 and 133 is toward thescreen of the TV 100, for example when the user makes a gesture ofreducing the distance between the two hands while stretching his bothhands 132 and 133, the controller 180 maintains the screen as is withoutrecognizing this as a gesture of zooming out the screen (S282).

However, if the travelling direction of the two hands 132 and 133 isopposite to the screen of the TV 100, for example when the user makes agesture of decreasing the distance between the two hands by pulling hishands 132 and 133 back, the controller 180 recognizes this as a gestureof zooming out the screen and performs a zoom-out operation (S283).

In the case of recognizing a user's gesture of increasing the distancebetween the two hands 132 and 133 after recognizing a user's gesture ofdecreasing the distance between the two hands 132 and 133 (S284), thecontroller 180 determines whether the travelling direction of the twohands 132 and 133 with respect to the screen of the TV 100 is toward thescreen of the TV 100 (S285).

If the travelling direction of the two hands 132 and 133 is toward thescreen of the TV 100, for example when the user makes a gesture ofincreasing the distance between the two hands while stretching his bothhands 132 and 133, the controller 180 maintains the screen as is withoutrecognizing this as a gesture of zooming in the screen (S286).

However, if the travelling direction of the two hands 132 and 133 isopposite to the screen of the TV 100, for example when the user makes agesture of increasing the distance between the two hands by pulling hishands 132 and 133 back, the controller 180 recognizes this as a gestureof zooming in the screen and performs a zoom-in operation (S287).

According to the method of recognizing the zoom-in operation shown inFIG. 14, even though the user decreases the distance between the twohands 132 and 133 by pulling his hands 132 and 133 back in a directionaway from the screen and then increases the distance between the twohands 132 and 133 by stretching the two hands 132 and 133, thecontroller 180 performs only the zoom-out operation but not the zoom-inoperation. According to an embodiment, a direction for determiningwhether to perform the zoom-in or zoom-out operation may be set oppositeto the direction considered in FIG. 14.

It has been described in connection with FIGS. 13 and 14 to neglect azoom-out gesture inevitably entailed in a zoom-in gesture and a zoom-ingesture inevitably entailed in a zoom-out gesture in consideration ofdirectivity of user's zoom-in and zoom-out operations. However, theembodiments of this document are not limited thereto, and other variousmethods may also be available to disregard unnecessary gestures.

For example, a method of considering the speeds of zoom-in and zoom-outgestures may be performed in order for the controller 180 to disregardunnecessary zoom-in and zoom-out gestures. Specifically, the controller180 may disregard unnecessary gestures by considering a difference inspeed between a zoom-in gesture and a zoom-out gesture entailed in thezoom-in gesture when a specific user attempts to zoom in the screen orby considering a difference in speed between a zoom-out gesture and azoom-in gesture entailed in the zoom-out gesture when the user attemptsto zoom out the screen.

Further, a method of considering a difference in time between a zoom-ingesture and a zoom-out gesture may be available in order for thecontroller 180 to disregard unnecessary zoom-in and zoom-out gestures.The controller 180 may disregard unnecessary gestures by considering adifference in time between a zoom-in gesture and a zoom-out gestureentailed in the zoom-in gesture when a specific user attempts to zoom inthe screen or by considering a difference in time between a zoom-outgesture and a zoom-in gesture entailed in the zoom-out gesture when theuser attempts to zoom out the screen.

FIG. 15 illustrates an exemplary user's gesture of zooming in the screenwhile a gesture user interface using the thumb 133B and index finger133A of user's right hand 133 is activated. Referring to FIG. 15, when adistance between the thumb 133B and the index finger 133A increases fromD1 to D2 by the user opening the thumb 133B and the index finger 133A(for example, from (a) to (b) in FIG. 15), the controller 180 of theelectronic device 100 may recognize this as a gesture of zooming in thescreen. However, the gesture is merely an example of a predeterminedzoom-in gesture, and the embodiments of this document are not limitedthereto. For example, other various gestures than the gesture ofincreasing the distance between the two fingers may be set as thezoom-in gesture.

FIG. 16 illustrates an exemplary user's gesture of zooming out thescreen while a gesture user interface using the thumb 133B and indexfinger 133A of user's right hand 133 is activated. Referring to FIG. 16,when a distance between the thumb 133B and the index finger 133Adecreases from D2 to D1 by the user closing the thumb 133B and the indexfinger 133A (for example, from (a) to (b) in FIG. 16), the controller180 of the electronic device 100 may recognize this as a gesture ofzooming out the screen. However, the gesture is merely an example of apredetermined zoom-in gesture, and the embodiments of this document arenot limited thereto. For example, other various gestures than thegesture of decreasing the distance between the two fingers may be set asthe zoom-out gesture.

The methods of disregarding unnecessary gestures considering directivityof zoom-in and zoom-out gestures or a difference in speed or timebetween zoom-in and zoom-out gestures as described above in connectionwith FIGS. 13 to 14 may also apply to a user interface using the thumb133B and the index finger 133A shown in FIGS. 15 and 16. This may beapparent to one of ordinary skill in the art based on those describedwith reference to FIGS. 15 to 16 and thus detailed description thereofwill be omitted.

FIG. 17 illustrates an example where a body part of a second user entersinto a virtual region set around a first body part of a first user whichis granted a first control right. Referring to FIG. 17, the first user130 is a person whose left hand 132 is granted the control right for theelectronic device 100, and the second user 130′ is a different person towhom no control right for the electronic device 100 is granted.

Referring to FIG. 17, under the circumstance where a spherical 3Dvirtual region is set around the left hand 132 of the first user 130that has been granted the first control right, the left hand 132′ of thesecond user 130′ enters into the virtual region. Depending on whetherthe first control right is maintained for the left hand 132 of the firstuser 130 and whether a control right is granted the left hand 132′ ofthe second user 130′, there may be a plurality of persons who areauthorized to control the electronic device 100 or the persons may bechanged to each other. This has been described above, and thus detaileddescription thereof will be omitted.

FIG. 18 illustrates an example where the first and second users 130 and130′ make a zoom-in gesture when a control right is additionally grantedthe left hand 132′ of the second user 130′ while the control right isgranted the left hand 132 of the first user 130.

Referring to FIG. 18, if the first and second users 130 and 130′increase a distance between their left hands 132 and 132′ from D1 to D2(for example, (a)→(b)), the controller 180 of the electronic device 100may recognize this as a gesture of zooming in the screen. However, thisis merely an example of a predetermined zoom-in gesture, and theembodiments of this document are not limited thereto.

FIG. 19 illustrates an example of a zoom-out gesture by the first andsecond users 130 and 130′ in a case where a control right isadditionally provided to the left hand 132′ of the second user 130′while a control right has been provided to the left hand 132 of thefirst user 130.

Referring to FIG. 19, if the first and second users 130 and 130′decrease a distance between their left hands 132 and 132′ from D2 to D1(for example, (a)→(b)), the controller 180 of the electronic device 100may recognize this as a gesture of zooming out the screen. However, thisis merely an example of a predetermined zoom-out gesture, and theembodiments of this document are not limited thereto.

The method of disregarding unnecessary gestures considering directivityof zoom-in and zoom-out gestures and difference in speed and timebetween zoom-in and zoom-out gestures as described in connection withFIGS. 13 to 14 may apply to the user interfaces shown in FIGS. 18 and 19as well. This is apparent to one of ordinary skill in the art from thosedescribed in connection with FIGS. 15 and 16, and detailed descriptionthereof will be omitted.

FIG. 20 illustrates an example of a process of determining whether firstand second body parts are included in the same user in the method shownin FIG. 9. According to FIG. 20, such determination is made based onwhether the first and second body parts are included in a skeleton-basedregion of the same user.

Referring to FIG. 20, the controller 180 of the electronic device 100sets a plurality of gesture points on body parts of a user. For example,since a gesture point GP1 corresponding to the left hand 132 of the user130 which is the first body part is connected to a gesture point GP2corresponding to the right hand 133 of the user 130 which is the secondbody part, the left hand 132 of the user 130 and the right hand 133 ofthe user 130 may be determined as body parts of the same user 130.

In a case where only the user 130 is included in an image obtained bythe camera 121, even though the obtained image is a 2D image, an erroris less likely to occur while the controller 180 determines whether thefirst and second body parts are included in the same user.

However, if a body part of the user 130 and a body part of another userare together included in the obtained image and the obtained image is a2D image, the likelihood of occurrence of an error increases while thecontroller 180 determines whether the first and second body parts areincluded in the same user.

For example, a reason may be because when a body part of the user 130and a body part of another user overlap each other in the obtainedimage, it is difficult to determine in which user the first and secondbody parts are included. Upon determining whether the first and secondbody parts are included in the same user, an analysis of depth dataobtained by a 3D camera may help the likelihood of occurrence of anerror to decrease.

FIG. 21 is a flowchart illustrating a method of controlling anelectronic device according to an embodiment of this document. It isassumed in FIG. 21 that first and second body parts are included in thesame user. Hereinafter, the method of controlling an electronic devicewill be described with reference to FIGS. 1 and 21.

The controller 180 of the electronic device 100 analyzes an imageobtained by the camera 121 to provide a first control right to a firstbody part of a user (S210) and sets a virtual region following the firstbody part (S220). Then, if an analysis result of the image obtained bythe camera 121 shows that a second body part of the user enters into thevirtual region (S230), the controller 180 determines whether the firstcontrol right provided to the first body part is maintained (S340).

If it is set to maintain the first control right, the controller 180 mayprovide a second control right to the second body part while maintainingthe first control right (S350). For example, the controller 180activates a gesture user interface using the first and second bodyparts.

However, if it is set to inactivate the first control right, thecontroller 180 inactivates the first control right and provides thesecond control right to the second body part (S360). For example, thecontroller 180 may change body parts used for a gesture user interfacefrom the first body part to the second body part.

Assuming that the first and second body parts are not included in thesame user unlike FIG. 21, the controller 180 may set a plurality ofpersons authorized to control the electronic device 100 or may changepersons authorized to control the electronic device 100 through stepS350 or S360.

FIG. 22 illustrates an example where a result of inactivation of agesture user interface is displayed on the electronic device 100according to the method illustrated in FIG. 21. For example, it isillustrated in FIG. 22 that while a left hand gesture user interface isactivated, the user's right hand enters into a virtual region, and whenthe right hand enters into the virtual region, a control right for theleft hand is inactivated.

Referring to FIG. 22, text and voice messages saying “right hand gestureUI mode has been activated” are outputted through the display 151 of theelectronic device 100 and the audio output module 152 of the electronicdevice 100, respectively. Although not shown in FIG. 22, the controller180 in this and other embodiments may output various text messagesand/or voice message corresponding to various settings of the gestureuser interface.

FIG. 23 illustrates a method of controlling an electronic deviceaccording to an embodiment of this document. For example, a process ofinactivating a control right provided to a user's body part is describedin connection with FIG. 23. Hereinafter, the method of controlling anelectronic device will be described with reference to FIGS. 1 and 23.

First, the controller 180 of the electronic device 100 provides firstand second control rights to first and second body parts of at least oneuser, respectively (S410). Then, the controller 180 continuouslyanalyzes an image of the user that has been captured by the camera 121(S420) and determines whether a predetermined gesture of abandoning acontrol right is recognized (S430).

If the predetermined control right abandonment gesture is recognized,the controller 180 inactivates at least one of the first and secondcontrol rights (S440). The predetermined control right abandonmentgesture may be changed depending on aspects of abandoning the providedcontrol rights. Some examples will be described with reference to FIGS.24 to 27.

FIG. 24 illustrates an example of a control right abandonment gestureaccording to the method illustrated in FIG. 23. A gesture of the userraising his hands 132 and 133 as high as his shoulders and then lowershands toward his waist may be preset as a gesture of abandoning both thefirst and second control rights. Upon recognizing the gesture of theuser 130, the controller 180 of the electronic device 100 may inactivateboth the first and second control rights provided to the two hands 132and 133.

FIG. 25 illustrates an example of displaying on the display 151 of theelectronic device 100 a result of abandonment of a control rightaccording to the control right abandonment gesture illustrated in FIG.24. Referring to FIG. 25, upon recognizing that the user 130 makes thegesture illustrated in FIG. 24, the controller 180 may output a messagesaying “control rights for both hands have been inactivated” through thedisplay 151 and the audio output module 152 of the electronic device100.

FIG. 26 illustrates an example of a control right abandonment gestureaccording to the method illustrated in FIG. 23. A gesture of the userraising his left hand 132 as high as his shoulders and then lowering theleft hand toward his waist may be assumed to be preset as a gesture ofabandoning the first control right. Upon recognizing the gesture of theuser 130, the controller 180 of the electronic device 100 may inactivatethe first control rights provided to the left hand 132.

FIG. 27 illustrates an example of displaying on the display 151 of theelectronic device 100 a result of abandonment of a control rightaccording to the control right abandonment gesture illustrated in FIG.26. Referring to FIG. 27, upon recognizing that the user 130 makes thegesture illustrated in FIG. 26, the controller 180 may output a messagesaying “control right for left hand has been inactivated” through thedisplay 151 and the audio output module 152 of the electronic device100.

FIG. 28 is a flowchart illustrating a method of controlling anelectronic device according to an embodiment of this document. It isassumed in FIG. 28 that the first and second body parts are included inthe same user, but as discussed above, they can belong to differentusers. Hereinafter, the method of controlling an electronic device willbe described with reference to FIGS. 1 and 28.

The controller 180 of the electronic device 100 analyzes an imageobtained by the camera 121 and provides a first control right to a firstbody part of a user (S510). Then, the controller 180 sets a fixed regionincluding the first body part as a virtual region at a specific time(S520).

Here, the virtual regions being the fixed region preferably means thatthe virtual region does not change its position even when the user movesthe first body part which is a basis of setting of the virtual region toanother position or location. Further, the time of setting of thevirtual region may be a time that the first control right is provided tothe first body part or a time that the controller 180 recognizes that agesture by the first body part corresponds to a predetermined gesturefor setting the virtual region.

As described in connection with FIG. 2, the controller 180 may releasesetting of the virtual region when a predetermined time has passed sincethe virtual region was set. The setting or release of setting of thevirtual region has been described above, and thus, repetitivedescription will be omitted.

If the virtual region is set, the controller 180 analyzes an imageobtained by the camera 121 and determines whether the first body partdeparts from the virtual region (S530). If the first body part departsfrom the virtual region, the controller 180 inactivates the firstcontrol right provided to the first body part (S540). However, if thefirst body part is within the virtual region, the controller 180maintains the first control right provided to the first body part(S550).

Then, the controller 180 analyzes an image obtained by the camera 121and determines whether the second body part of the user enters into thevirtual region (S560). If the second body part enters into the virtualregion, the controller 180 may provide the second control right to thesecond body part.

If the first control right is inactivated according to step S540, thecontroller 180 activates a gesture user interface using the second bodypart. However, if the first control right is maintained according tostep S540, the controller 180 activates a multi-gesture user interfaceusing the first and second body parts.

In a variation, if it is assumed that the first and second body partsare included in different users, respectively, then the controller 180may set a plurality of persons authorized to control the electronicdevice 100 or may change persons authorized to control the electronicdevice 100 through steps S530 to S570.

FIG. 29 illustrates an example of a virtual region setting gestureaccording to the method illustrated in FIG. 28. Referring to FIG. 29, auser of the electronic device 100 may set a virtual region including theleft hand 132 by lifting his left hand 132 that has been positioned tobe parallel with his shoulders beyond his head. This gesture is merelyan example of a gesture of setting the virtual region, and theembodiments of this document are not limited thereto.

If the above gesture is a gesture to provide a control right for theelectronic device 100 to the left hand 132, the virtual region may beset while a control right is simultaneously provided to the left hand132. Further, the user may also set a virtual region for changing thesettings of a gesture user interface by making the above gesture apredetermined time after the control right has been provided to the lefthand 132.

FIG. 30 illustrates a process of changing control rights of a gestureuser interface according to the method illustrated in FIG. 28. Referringto (a) of FIG. 30, while the first control right is provided to theuser's left hand 132 and a 3D virtual region which is a spherical fixedregion and includes the left hand 132 is set, the user moves his lefthand 132 outside of the virtual region and moves his right hand 133inside of the virtual region. The first control right provided to theleft hand 132 may still be maintained and the second control right isnot yet provided to the right hand 133.

Referring to (b) of FIG. 30, the left hand 132 departs from the virtualregion, and the right hand 133 enters into the virtual region. Thecontroller 180 may inactivate the first control right provided to theleft hand 132 and may provide the second control right to the right hand133. For example, as shown in FIG. 30, the controller 180 may changebody parts used for the gesture user interface for the electronic device100 from the left hand 132 to the right hand 133.

FIG. 31 illustrates an example of displaying a result of change ofcontrol rights of the gesture user interface by the gesture illustratedin FIG. 30. Referring to FIG. 31, the controller 180 of the electronicdevice 100 may output a message saying “control right for left hand hasbeen inactivated and control right has been provided to right hand”through the display 151 and the audio output module 152.

FIG. 32 illustrates an example where if a both-hand gesture userinterface is activated, a region including a virtual region set aroundthe left hand 132 granted with a control right is set as a gesturerecognition region. Referring to (a) of FIG. 32, the right hand 133enters into the virtual region set around the left hand 132 granted withthe control right. Then, the controller 180 of the electronic device 100releases setting of the virtual region and sets the gesture recognitionregion to include the space of the virtual region around both hands 132and 133 (refer to (b) of FIG. 32).

FIG. 33 illustrates an example where when a both-hand user interface isactivated, a region including a virtual region set around a left hand132 granted with a control right is set as a gesture recognition region.Referring to FIG. 33, if the right hand 133 enters into a virtual regionset around the left hand 132, the controller 180 of the electronicdevice 100 sets a 3D space which includes the virtual region and wherethe two hands 132 and 133 may be positioned as a gesture recognitionregion.

FIG. 34 illustrates an example where when a user's body part granted acontrol right departs from a recognition range of the camera 121, thecontrol right granted to the body part is lost. Referring to (a) of FIG.34, the left hand 132 of the user 200 which is granted a control righthas departed from a recognition range of the camera 121 mounted on theTV 100. Then, the controller 180 of the TV 100 may relinquish thecontrol light from the left hand 132.

Referring to (b) of FIG. 34, among the thumb 133B and index finger 133Agranted with control rights, the thumb 133B departs from a recognitionrange of the camera 121 mounted on the mobile terminal 100. Then, thecontroller 180 of the mobile terminal 100 may relinquish the controlright from the thumb 133B, and the mobile terminal 100 may be controlledby a gesture user interface using the index finger 133A.

The above-described methods according to the embodiments of thisdocument may be implemented as programs that may be executed by variouscomputer means and recorded in at least one computer-readable medium.The computer-readable medium may contain a program command, a data file,and a data structure, alone or in a combination thereof. The programrecorded in the medium may be one specially designed or configured forthe embodiments of this document or one known to those of ordinary skillin the art.

Examples of the computer-readable medium may include magnetic media,such as hard disks, floppy disks, or magnetic tapes, optical media, suchas CD-ROMs or DVDs, magneto-optical media, such as floptical disks,ROMs, RAMs, flash memories, or other hardware devices that areconfigured to store and execute program commands. Examples of theprogram may include machine language codes such as those made by acompiler as well as high-class language codes executable by a computerusing an interpreter. The above-listed hardware devices may beconfigured to operate as one or more software modules to perform theoperations according to the embodiments of this document, and viceversa.

According to the embodiments of this document, the electronic device andthe method of controlling the electronic device may easily and rapidlyactivate a gesture user interface using a plurality of objects and mayeasily and rapidly control the control rights provided to the pluralityof objects.

The invention has been explained above with reference to exemplaryembodiments. It will be evident to those skilled in the art that variousmodifications may be made thereto without departing from the broaderspirit and scope of the invention. Further, although the invention hasbeen described in the context its implementation in particularenvironments and for particular applications, those skilled in the artwill recognize that this document's usefulness is not limited theretoand that the invention can be beneficially utilized in any number ofenvironments and implementations. The foregoing description and drawingsare, accordingly, to be regarded in an illustrative rather than arestrictive sense.

1-33. (canceled)
 34. An electronic device comprising: a controller thatassigns a first control right to a first object included in an imageobtained from a camera, sets a virtual region including the firstobject, and assigns a second control right to a second object when thesecond object enters into the virtual region, wherein each of the firstand second control rights is a right to control an operation of theelectronic device by a gesture of at least one of the first and secondobjects.
 35. The electronic device of claim 34, wherein the virtualregion is a predetermined area surrounding the first object, and followsthe first object when the first object moves.
 36. The electronic deviceof claim 34, wherein the virtual region is a predetermined area fixed inspace with respect to the camera, and does not move when the firstobject moves.
 37. (canceled)
 38. The electronic device of claim 34,wherein when the second control right is assigned to the second object,the controller enters the electronic device into a multi-object mode andtracks a movement of both the first and second objects to control theoperation of the electronic device.
 39. The electronic device of claim34, wherein when the second control right is assigned to the secondobject, the controller enters the electronic device into a single objectmode by relinquishing the first control right from the first object, andtracks a movement of the second object to control the operation of theelectronic device.
 40. (canceled)
 41. (canceled)
 42. The electronicdevice of claim 34, wherein when the second object enters into thevirtual region, the controller provides the second control right to thesecond object while maintaining the first control right provided to thefirst object.
 43. The electronic device of claim 34, wherein when thesecond object enters into the virtual region, the controller inactivatesthe first control right provided to the first object while providing thesecond control right to the second object.
 44. (canceled)
 45. (canceled)46. The electronic device of claim 34, wherein when the first controlright is provided to the first object, the controller sets apredetermined region surrounding the first object as the virtual region,or wherein when the first object is moved according to a predeterminedtrajectory, the controller sets the virtual region based on thepredetermined trajectory.
 47. The electronic device of claim 34, whereinwhen the second control right is provided to the second object, thecontroller sets a gesture recognition region including at least thevirtual region.
 48. The electronic device of claim 34, wherein thecontroller releases setting of the virtual region when a predeterminedtime has passed after the virtual region was set or when the secondobject enters into the virtual region within the predetermined timeafter the virtual region has been set.
 49. (canceled)
 50. (canceled) 51.A method for controlling an electronic device including a controller,the method comprising: assigning, by the controller, a first controlright to a first object included in an image obtained from a camera;setting a virtual region including the first object; and assigning asecond control right to a second object when the second object entersinto the virtual region, wherein each of the first and second controlrights is a right to control an operation of the electronic device by agesture of at least one of the first and second objects.
 52. The methodof claim 51, wherein the virtual region is a predetermined areasurrounding the first object, and follows the first object when thefirst object moves.
 53. The method of claim 51, wherein the virtualregion is a predetermined area fixed in space with respect to thecamera, and does not move when the first object moves.
 54. The method ofclaim 51, further comprising: when the second control right is assignedto the second object, entering the electronic device into a multi-objectmode, and tracking a movement of both the first and second objects tocontrol the operation of the electronic device.
 55. The method of claim51, further comprising: when the second control right is assigned to thesecond object, entering the electronic device into a single object modeby relinquishing the first control right from the first object, andtracking a movement of the second object to control the operation of theelectronic device.
 56. (canceled)
 57. (canceled)
 58. The method of claim51, wherein when the second object enters into the virtual region, thestep of assigning the second control right includes: providing thesecond control right to the second object while maintaining the firstcontrol right provided to the first object.
 59. The method of claim 51,wherein when the second object enters into the virtual region, the stepof assigning the second control right includes: inactivating the firstcontrol right provided to the first object while providing the secondcontrol right to the second object.
 60. (canceled)
 61. (canceled) 62.The method of claim 51, further comprising: when the first control rightis provided to the first object, setting a predetermined regionsurrounding the first object as the virtual region; or when the firstobject is moved according to a predetermined trajectory, setting thevirtual region based on the predetermined trajectory.
 63. The method ofclaim 51, further comprising: when the second control right is providedto the second object, setting a gesture recognition region including atleast the virtual region.
 64. The method of claim 51, furthercomprising: releasing setting of the virtual region when a predeterminedtime has passed after the virtual region was set or when the secondobject enters into the virtual region within the predetermined timeafter the virtual region has been set.
 65. (canceled)